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Assessing and acknowledging learning through non-accredited community 
adult language, literacy and numeracy programs 
by Darryl Dymock and Stephen Billett, Griffith University 



Beyond the obvious acquisition of the three ‘R's’ — reading, writing and arithmetic — many other benefits 
derive from learning. Increases in self-esteem and confidence have often been cited as consistent, if 
unanticipated, outcomes of learning and have been achieved even when there has been limited gain in 
the competency being learned. Further; such confidence-building can have a positive impact on how and 
what an individual learns. While the measurement of objective competencies acquired through learning 
is well developed, the measurement of the wider benefits of learning is still in its infancy. 

This report by Darryl Dymock and Stephen Billett identifies the extent to which indicators might be 
developed for a range of learning outcomes in non-accredited community adult language, literacy and 
numeracy programs. It follows Dymock's earlier work, Community adult language, literacy and numeracy 
provision in Australia (NCVER, 2007), which attempted to gauge the extent of provision of this type of 
training in Australia. 

Key messages 

■ It is possible to assess the wider benefits of non-accredited adult language, literacy and numeracy 
learning. However; different types of assessment instruments, which cater for the diversity and 
complexity of learners' needs, motivations and outcomes, are necessary. Special attention must be 
paid to the language adopted in the instruments so that these can be easily used by tutors and 
their students. 

■ Both learners and tutors derive personal and educational outcomes by participating in the process 
of assessing and acknowledging learning outcomes. 

■ Tutors in this field, many of whom are volunteers, are in need of professional development to fully 
understand the purpose and language of the assessment instruments. 

■ The right approach to assessment can build the self-confidence of students, many of whom are not 
suited to formal education settings. 



Tom Karmel 

Managing Director; NCVER 



Informing policy and practice in Australia’s training system . . . 
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Executive summary 



Purpose 

This research explores the potential for developing instruments that assess and acknowledge the 
wider benefits of learning for adults participating in non-accredited community language, literacy 
and numeracy programs. The extent and variety of the learning needs and motivations of 
participants in these programs are not yet sufficiently accounted for in other assessment 
frameworks, such as the National Reporting System (NRS). 1 Therefore, it is necessary to identify 
approaches and procedures for assessing and acknowledging the outcomes that participants achieve 
in these courses. For this study, ‘non-accredited’ learning is that for which no nationally accredited 
vocational education and training (VET) qualification is awarded to learners. 



Approaches 

The research comprised three stages. Firsdy, a review of national and international research was 
undertaken to map the wider benefits of learning that have been identified in non-accredited adult 
learning. This review was augmented by an analysis of data collected in a 2006 study that mapped 
the provision of non-accredited community learning in adult language, literacy and numeracy in 
Australia (Dymock 2007a). 

Secondly, following that review, a number of potential assessment instruments that could be used 
to assess and acknowledge the outcomes of learners in non-accredited adult language, literacy and 
numeracy programs were identified by the authors. Through collaboration with five research 
partners, six of these instruments were selected as being most appropriate for these purposes. 2 The 
research partners were community education providers in Queensland, Victoria, South Australia 
and Western Australia and were selected for reasons of diversity in user groups and locations. 

Thirdly, the selected instruments were trialled and evaluated in the field at the partner sites. After 
the trial period, partner representatives reported on their experiences in using the instruments to 
assess and acknowledge wider learning outcomes in non-accredited adult language, literacy and 
numeracy. 



1 The National Reporting System (NRS) is a mechanism for reporting the outcomes of adult English language, literacy 
and numeracy provision in the VET system in labour market programs and in the adult and community education 
(ACE) sector. The National Reporting System takes into account the complexity of language, literacy and numeracy 
competence, considering not just competence with linguistic and mathematical systems, but also the ability to shape 
language and numeracy use according to purposes and contexts. The National Reporting System has been recently 
reviewed. It will be known as the Australian Core Skills Framework and will be released in late 2008. 

2 The selected instruments were: (i) Changes in confidence; (ii) Learner outcomes: personal, social, economic; (iii) 
Individual learner profile; (iv) Learner indicators of success; (v) Criteria for achievement; (vi) Skills and wider outcomes. 
Each of instruments (i) to (v) was based on existing measures and adapted by the participating research partners to suit 
local needs. Instrument (vi) is a composite instrument developed by the authors from their review of research and the 
interview responses. The ‘Findings’ section provides further information on each of the selected instruments. A copy 
of each instrument can be found in appendix 2. 
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Findings 

An understanding of the full extent of the learning arising through learners’ participation in non- 
accredited language, literacy and numeracy programs takes account of the following key factors: the 
purpose or motivation for learning; the processes of assessment, if it occurs at all; and the 
competence of voluntary tutors. As a result, the study found that a range of instruments and 
approaches would probably be required to assess and acknowledge learning outcomes that are 
specific to particular learners or cohorts of learners. 

Diverse needs 

People who participate in these programs range from learners with significant disabilities, to those 
whose language development has never been fully realised, through to those developing their 
language competence because of social, family or work-related necessity. Other reasons for 
participating include changes in attitudes and wanting to develop a greater sense of self and 
personal competence (that is, being confident enough to accept new challenges). Such diversity of 
learner and learning goals warrants assessment processes commensurate with these goals. 

Diverse outcomes 

From the review of literature the research identified seven categories of ‘wider benefits’ of learning. 
These were: self-confidence; engagement with others; attitudes to learning; agency/ pro-activity 
(actively learning through experience); life trajectories; personal growth; and social capital. This 
study confirmed those categories and the importance of acknowledging the wider benefits of 
learning. In general, these benefits include enhanced personal confidence, heightened social 
competence, and the opening-up of possibilities and prospects that were not options prior to 
participation in these programs. Outcomes such as these are not the focus of or are captured in 
accredited learning frameworks like the National Reporting System. Importantiy, the findings of 
this review also pointed to the development of a ‘learning identity’ as a key element of engagement 
in the learning process, a concept that needs further research. 

Diverse kinds of instruments 

Given the diversity of learner outcomes arising from non-accredited language, literacy and 
numeracy programs, the approaches to assessment need to be correspondingly diverse to ensure 
that the wider benefits of learning are captured. In the practical component of the research, all six 
instruments were used to some effect in different teaching and learning contexts. However, no 
single instrument was identified as being the most preferred across all sites. Instead, each of the 
instruments in different ways had particular qualities and uses, which underlined that these 
assessment processes are likely to require a range of instruments and usages. 

The instruments were most useful when the level of language used to capture the learning was easy 
for the learners to understand — or at least for the tutors to interpret for them. Moreover, the 
relative value given to each of the wider benefits will vary at a local level, and, consequently, so will 
the choice of indicators. A ‘portfolio’ of instruments is therefore likely to be necessary to address 
diversity in learners’ and tutors’ needs. In other words, greater consideration needs to be given to 
the processes involved in assessing and acknowledging learner outcomes. The simple adoption of 
uniform measures is unlikely to prove satisfactory or be appropriate. 

How such instruments are used instead of or in conjunction with existing frameworks is a decision 
for those responsible for the programs, but their administration has the potential to be particularly 
useful in formative as well as summative learner assessment. External reporting (for example, to 
management or a funding body) may best be done through the aggregation of individual 
assessments of outcomes, the nature of this summary depending on the instrument used. 
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Role of tutors 



The instruments trialled were used to identify individual achievements; they also enabled tutors to 
identify areas of strengths and weaknesses in those achievements, this process informing 
subsequent learning and assessment activities. The assessment processes found to work best were 
those that comprised a joint activity between learner and tutor. Moreover, there are potential 
benefits for both learner and tutor from engagement in processes that extend beyond what is 
captured by the instrument. However, like the learners, the largely volunteer tutors in these 
programs have different levels of competence and readiness to assess these kinds of learning 
outcomes. They possess diverse levels of language and literacy and instructional and assessment 
competence. So, training tutors to understand the purposes and language of the instruments is 
likely to be essential. 



Conclusion 

The importance of establishing assessment measures and approaches commensurate with the 
diversity of learner needs, motivations and outcomes in non-accredited language, literacy and 
numeracy programs is a key finding of this study. Allied to this is the value of tutors engaging 
directly with participants in completing these assessment instruments. This study has shown that it 
is possible to develop and use instruments to assess and acknowledge the benefits of learning 
through these programs; the benefits are of the kind that goes beyond the outcomes addressed by 
the National Reporting System. Indeed, the diversity of outcomes necessitates having access to a 
range of instruments from which to select, in order to cater not only for the learners’ individual 
needs, but also for the abilities and preferences of tutors and coordinators. 

The findings also identify potential benefits for both learners and tutors from joint engagement in 
the assessment process. However, tutors may need training to better understand and use the 
instruments available to them. The instruments also need to be accessible and comprehensible to 
the learners, particularly those who are second language learners. 

While this study has highlighted the diversity of purposes, processes and outcomes from non- 
accredited community language, literacy and numeracy training, it also draws attention to the 
potential for developing a national scheme for reporting on the wider outcomes of this type of 
learning. However, the desirability and practicality of developing and implementing such a reporting 
scheme warrants further investigation. At the individual level, the development of ‘learner identity’ 
as part of engaging in the learning process also needs further exploration in non-accredited adult 
learning generally, because this appears to be a key factor in engaging disengaged and disadvantaged 
adults in learning and helping them into more active, productive and satisfying lives. 
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Introduction 



A recent study of non-accredited language, literacy and numeracy provision in Australia (Dymock 
2007a) found that the perceived benefits of program participation were diverse for learners and 
included the development of language, literacy and numeracy skills and personal growth. 

Participants also gained confidence to engage in further education and training or in employment. 
That report concluded that, in non-accredited adult language, literacy and numeracy provision in 
Australia, the ‘development of language, literacy and numeracy skills goes hand-in-hand with 
development of self-confidence’ (p.35). So, beyond the development of literacy and numeracy skills, 
other important personal learning outcomes might be achieved. 

This confirms findings in other adult literacy research (for example, Ward & Edwards 2002; 

Westell 2005) and experiences in the wider field of adult and community education (ACE). For 
example, the Adult Community and Further Education Board (2006, p.3) in Victoria stated that 
non-accredited learning is not only a pathway into accredited courses, but also a means for 
‘building confidence, resilience and self worth, enabling learners to make connections with family 
and the wider community’. Similarly, a British study linking learning and confidence development 
noted that: 

In non-accredited learning, the challenge is to clearly capture individual and group 
achievements, in order to demonstrate to learners, tutors and funders what has been gained. 

If gains in confidence are as significant as many learners and tutors appear to suggest, ways of 
evidencing them seem to be important. (Eldred et al. 2004, p.4) 

Currentiy, such important learning outcomes are not effectively accounted for in existing 
assessment regimes, including the National Reporting System. In non-accredited language, literacy 
and numeracy in Australia, Dymock identified that ‘ways of evidencing’ varied considerably. From a 
national survey of 125 providers and seven case studies in three states, he concluded that: 

. . . assessment of student learning, particularly among those using volunteers one to one is 
not especially rigorous. On the other hand, the range of motivations identified in the survey 
and in the interviews suggests that a more rigorous approach to assessment may not be 
appropriate. And if, for example, the development of self-confidence is seen as a worthwhile 
outcome, how might that aspect be assessed? (Dymock 2007a, p.33) 

The present project further explores this concern. That is, if the development of language, literacy 
and numeracy skills is concurrent with growth in learners’ confidence, the assessment of outcomes 
in non-accredited programs needs to find ways of assessing and recognising this learning, as well as 
supporting it. Consequendy, the focus here is restricted to non-accredited learning; that is, that for 
which no nationally accredited vocational education and training (VET) qualification is awarded to 
learners, and including courses or learning activities for which statements of attainment or 
statements of attendance are given. 

To date, no widely acceptable instrument has been developed or adopted in the non-accredited 
language, literacy and numeracy field in Australia that captures these wider kinds of learning 
outcomes. The National Reporting System (Department of Education, Science and Training 2007), 
soon to be revised and renamed 3 , is specifically designed to measure levels of competence in five 



3 The revised National Reporting System, the Australian Core Skills Framework, is due for release in the near future. 
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skill areas: reading, writing, oral communication, learning strategies and numeracy. Absent here are 
those outcomes associated with other areas of learner development and identified in studies from 
Australia and overseas. Also, as a competency-based approach is adopted for the National 
Reporting System, which by definition and practice focuses on measurable performances, it is not a 
wholly appropriate approach for identifying or recognising outcomes from learning that cannot be 
easily measured. 

The only systematic attempt in Australia to acknowledge wider achievements in non-accredited 
learning, as well as pathways to other options, appears to be the ‘A-frame’, endorsed by the 
Victorian Adult and Further Education Board in 2006. The A-frame, ‘an ACE framework for non- 
accredited learning building on the best’, comprises eight forms for ‘considering and documenting a 
learner’s goals and outcomes and for recording and monitoring learner feedback’ (Adult and 
Further Education Board 2006, p.2). One form provides for ‘Pathways and options — where 
learning leads’ and includes such aspects as work, further education, improved confidence, and 
attitude change. Flowever, this description is given as part of a tutor’s program planning process, 
rather than for monitoring individual outcomes. The learner feedback form allows the learner to 
comment on the learning process and on what they plan to do with the new knowledge. The A- 
frame might be seen as a precursor to the sorts of instruments explored here, except that the latter 
are in a specific language, literacy and numeracy context. 

The purpose of this project is not to try to fit existing frameworks to learning outcomes, but to 
identify the extent to which indicators might be developed for the range of learning outcomes 
achieved by learners in adult language, literacy and numeracy programs. No doubt there are 
examples of such indicators being used by practitioners at a local level. Flowever, this project 
applies the findings from national and international research to a number of language, literacy and 
numeracy programs across several states in order to identify potentially useful approaches. It does 
not attempt to validate the instruments used, nor to develop a particular instrument for national 
use, but to explore the potential of such developments for Australian programs. Such 
considerations might also inform any subsequent revision of the successor to the National 
Reporting System. 
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Approach 



Purpose 

The purpose of this research is to explore the potential for developing an instrument or 
instruments capable of assessing and acknowledging the wider benefits of learning for adults 
participating in non-accredited community language, literacy and numeracy programs. Such 
instruments need to take into account the extent and variety of their learning needs and their 
motivations and readiness. Four research questions were developed to guide that exploration: 

1 What are the main language, literacy and numeracy learner groups and their needs in non- 
accredited programs? 

2 What are the main outcomes from non-accredited language, literacy and numeracy programs, 
and what is the role of personal agency and goals in achieving these outcomes for individual 
learners? 

3 How might these outcomes best be ‘measured’ in order to indicate the extent of progress that 
learners are making, in terms of reporting against existing frameworks and the personal needs 
and interests of participants? 

4 To what extent might such indicators of progress be appropriate for incorporation into a 
national reporting process? 



Methodology 

This research project comprised three stages: a review of existing research; the development of a 
portfolio of possible instruments for monitoring learner outcomes; and the trialling of those 
instruments in non-accredited literacy and numeracy programs. 

The review of research considered existing literature focusing on assessment practices in non- 
accredited adult and community education and language, literacy and numeracy. The literature was 
both international and Australian in focus and aimed to identify the strengths and weaknesses of 
practices. Literature on personal agency and learner identity was also reviewed to identify those 
aspects most relevant to adult language, literacy and numeracy learning. In addition, data collected 
from the 2006 national mapping exercise of non-accredited language, literacy and numeracy 
(Dymock 2007a) were interrogated in order to inform the proceedings and derive findings 
particularly relevant to this project. 

The process for developing a portfolio of possible instruments comprised the identification of research 
partners, undertaking semi-structured interviews and discussions with coordinators, teachers and 
learners at the partner sites, and working collaboratively with the partners to assemble a tentative 
‘portfolio’ of assessment instruments designed to monitor wider learning outcomes in non- 
accredited adult language, literacy and numeracy programs. 

The research partners were chosen from responses to an invitation (see support document) sent to 
59 organisations in Western Australia, South Australia, Victoria and Queensland which had identified 
themselves in Dymock’s study (2007a) as community providers of adult language, literacy and 
numeracy. Five partners were chosen, based primarily on the need for a variety of provider types 



12 



Assessing and acknowledging learning through non-accredited community adult LLN programs 




and client groups, and for metropolitan and rural representation. The partners were: Pine Rivers 
Neighbourhood Centre, Lawnton, Queensland (semi-rural); Glenroy Neighbourhood Learning 
Centre, Victoria (urban); Hackham West Community Centre, and Aberfoyle Community Centre, 
South Australia (urban); and Read Write Now! (RWN), Western Australia (two sites: one rural, one 
urban). A profile of each of the partner organisations is included in the support document. 

The purpose of the interviews was to gather data that would permit the development of assessment 
instruments that would meet both individual and institutional imperatives for determining 
outcomes. Copies of the interview schedules are included in the support document. The breakdown 
of the interviews at the six partner sites is shown in table 1. 



Table 1 Interviews and other data sources, by partner and category 



Partner 


Coordinator 


Teachers 


Tutors 


Learners 


Other 


Total 


Aberfoyle Community 
Centre, SA 


1 


1 


5* 


3 


Community 

Development 

Officer 


11 


Glenroy Neighbourhood 
Learning Centre, Vic. 


1 


2 




6 


Written feedback 
from coordinators’ 
meeting with 4 
teachers/ tutors 


13 


Hackham West Community 
Centre, SA 


1 




5* 


4 




10 


Pine Rivers Neighbourhood 
Centre, Qld 


1 




5* 




1 team leader — 

community 

learning 


7 


Read Write Now!: 
Northam/Toodyay, WA 


1* 




3* 


2 




6 


Read Write Now! Perth 
urban/regional 


1 




3* 


5 


1 program 
manager, 

1 learner survey 
interviewer 


11 


Read Write Now! 
Coordinators’ conference, 
Perth 










35 coordinators 
(group)* 


35 


Total 


6 


3 


21 


20 


43 


93 



Note: * Work in voluntary capacity 



Table 1 indicates that data were gathered from a total of 93 people in four states. A feature of non- 
accredited adult language, literacy and numeracy provision is the heavy reliance on volunteers and 
part-time paid staff. As indicated in table 1, 57 (78%) of the teachers, tutors and coordinators 
informing this research project were volunteers. 

Trialling the instruments in the field was undertaken over a period of eight weeks. Six trial instruments 
were sent to the coordinators at the partner sites, with a proviso that they could amend individual 
instruments for local use, as well as create their own to add to the portfolio (see appendix 2). The 
coordinators were asked to document the feedback from tutors and learners on the instruments 
selected, along with justification for any changes made or new instruments developed, and to report 
on these at a national workshop. Those reports provide the main basis for the discussion section 
later in the report. 
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Findings 



This section reports, in turn, the findings from the three strands of research: the review of existing 
research; the development of a portfolio of possible instruments for monitoring learner outcomes; 
and the trial of those instruments in the field. A full account of the review of research is included in 
the support document. 



Review of research 

It was important initially to identify the nature of the participants in these programs, their reasons for 
participating and their readiness to participate. In the 2006 survey mentioned above, 64 of the 125 
respondents were identified as community providers on the basis that they nominated their primary 
role as either ‘General adult/ community education’, ‘Specific adult literacy/ numeracy improvement’, 
or ‘Community information/referral’ (Dymock 2007a). For the current study to further clarify the 
nature of the client groups and, in particular, their learning needs amongst these providers, the 
responses to selected items from the 2006 survey for those 64 providers were re-analysed. 

It was found that approximately two-thirds of the learners are women, and the largest group is in 
the 30—49 years age bracket, the prime family-rearing and income-earning years, which may 
influence learning goals and life trajectories. The next largest is the 50—59 years age group, also an 
active earning period for many adults. However, the perception of one-third of the providers is that 
the biggest motivation for learners in such programs is a general need to improve their language 
and literacy skills for daily use rather than for any specific needs such as employment. But 
employment- related needs are still strong and perceived to be relevant to learners by about a 
quarter of the community adult language, literacy and numeracy providers who responded. Overall, 
general and person-oriented needs (for example, social interaction and developing self-esteem) 
comprise some 60% of the reported perceived reasons for participation in these programs. For the 
purposes of this study, the breadth of ages and diversity of needs suggest that any instruments used 
to monitor outcomes have to allow for a range of individual purposes and a variety of outcomes. 
Importantly, this includes those associated with personal growth, enhanced self-confidence and 
capacities for greater social interaction. 

The concept of ‘wider learning outcomes’ is pervasive in the literature, particularly in language, 
literacy and numeracy learning. For example, a Tasmanian adult literacy survey (Department of 
Education, Tasmania 2005, p.12) identified 11 ‘other outcomes’, including pathways to further study 
and employment, better skills in researching and finding information, and a significant increase in 
self-esteem and in opportunities to participate in the community. A United Kingdom study of adult 
learners’ lives (Barton et al. 2004, p.101) identified the ‘wider benefits of learning’ as empowerment 
in the classroom and in life, new skills acquisition, and a change in attitudes to learning, which 
added quality to life. Ward and Edwards concluded from their research with literacy and numeracy 
learners in north-west England, which used the ‘learning journey’ as a metaphor, that: 

Perhaps the most profound change for most learners interviewed was a massive enhancement 
of their confidence and self-esteem. This increased confidence had a significant impact on 
their learning achievements, attitudes to learning, aims and aspirations, ability to do real life 
activities and their social activities with other people. (Ward & Edwards 2002, p.4) 
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Similarly, a study of adult numeracy learners in England (Swain 2006) found that almost three- 
quarters reported they had changed as a person in some way through learning maths, and that some 
learners changed their aspirations, as their sense of achievement and level of self-esteem grew. 

Swain concluded that: 

Although human agency may be fragile, particularly for those with little power, the students in 
this study were still able to make decisions that had the potential, at least, to be able to re- 
direct their lives. (Swain 2006, p.3) 

In a review of United States literature on ‘non-academic’ outcomes of adult literacy programs, 
Westell (2005) identified five main outcomes: self-confidence, independence, attitude change, 
relationship and community building, and learning to learn. Westell stated that every study 
mentioned self-confidence as being crucial to ongoing learning and development. 

Eldred at al (2004, p.57) concluded that further research ‘should be carried out to develop best 
practice in framing confidence growth as a learning outcome and methods of recognizing and 
recording in this area’. This current study is intended to contribute to that work. We are conscious, 
however, of Grief and Windsor’s warning that: 

We need to be cautious that the desire to make systems robust and ensure the credibility of 
data on learners’ achievements does not encourage the development of systems that resemble 
that of award schemes and qualifications. By doing this, we lose the very features of non- 
accredited learning for which this option is currently valued. (Grief & Windsor 2002, p.63) 

Consequently, this research project resisted trying to fit existing competency-based frameworks, 
such as the National Reporting System, to the range of learning outcomes from non-accredited 
language, literacy and numeracy programs. Instead, it was decided to consider the outcomes broadly 
in order to explore the extent to which a complementary framework might be developed to assess 
and acknowledge the breadth of learning occurring in these programs. 



Possible bases for revised approaches for monitoring learner outcomes 

Drawing on the findings reported above, a number of possible bases for assessing and 
acknowledging the wider benefits of learning in non-accredited community language, literacy and 
numeracy, in terms of growth of confidence and related attributes, were identified. These included: 

self-confidence and personal competence-, the extent to which the learner has a sense of self and a belief 
in being able to put their capabilities into action 

engagement with others-, the extent of the learner’s interaction with other individuals in the family, at 
work and at sites such as schools, government offices, and shops 

•y" attitudes to learning, the extent of the learner’s attitude towards current and future learning, and 
their ability to learn how to learn 

■y- agency/ pro-activity, the extent to which the learner actively accesses and negotiates with and learns 
from experience and is able to shape construction of that experience 

life trajectories-, the extent of the learner’s goals and ambitions and expectations of where life will 
lead 

-j- personal growth / personal change-, the extent to which a learner perceives they have grown and/ or 
changed as a person 

social capital-, the extent of community participation, involvement in networks, clubs and other 
social situations. 

These potentially useful bases for assessing and acknowledging learning in these programs were 
subsequently explored through interviews and discussions with the research partners in the next 
phase of the project to develop assessment instruments for trialling, along with confirmation of the 
needs of learners. 
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Collaborative identification of potential instruments 

Confirming learner needs 

Discussion with learners, coordinators, teachers and tutors confirmed the diversity of learner needs 
identified in the literature review. For example, one learner told of her embarrassment and 
indignation when a gardener was brought in to interpret when her husband was admitted to 
hospital, and how that incident finally motivated her to join an English class. Another learner said: 

I was . . . with Centrelink and I . . . had to do a big group class of improving your skills to look 
for work and write resumes and read papers and all that. And I just blew up under pressure 
and couldn’t understand and wouldn’t even let myself try to understand. I just ran out of the 
room . . . and my room teacher . . . said ‘. . . you obviously need one-on-one to help you 
understand things a bit more.’ 

The motivations expressed by learners included a desire to be able to learn English at a mature age 
and to improve opportunities through higher levels of educational outcomes. Moreover, learners 
wished to overcome personal limitations that constrain work and/ or life options, to participate in 
Australian higher education (for a new arrival in this country), and to have the capacity to play a 
more central role in family life. 

Coordinators of non-accredited language, literacy and numeracy programs provided examples of 
learners with disabilities, learners who had very negative school experiences, those who left school 
early, or who lacked self-esteem. Coordinators commented that some had short-term needs, such as 
getting a driver’s licence, or improving their English for accredited courses or for employment 
purposes; others need long-term support to achieve their personal goals. 

A variety of individual needs and motivations were revealed — personal, social and vocational — 
confirming the findings from the literature. 



Testing the possible bases for identifying wider benefits of learning 

Coordinators, teachers, tutors and learners were specifically asked to identify wider learning outcomes, 
that is, beyond the development of language, literacy and numeracy competence. Their responses 
were then categorised under each of the seven concepts identified from the literature review as 
possible bases for indicating the wider benefits of learning: self-confidence; engagement with others; 
attitudes to learning; agency/ pro-activity; life trajectories; personal growth; and social capital. Further 
details and illustrative examples of these responses are provided in the support document. 

The examples obtained through the interviews show the extent of the interrelatedness of the seven 
proposed bases for assessing these wider benefits and the recurrence of certain general outcomes: 
improved personal capacity and competence; empowerment, responsibility and autonomy; greater 
engagement with others and in the community, and a new capacity for perceiving the world and 
their place in it. Underpinning every one of these bases is learner confidence. 

What was also evident from this research phase was that any attempt to monitor learner outcomes 
in non-accredited language, literacy and numeracy programs must take into account a wide range of 
possible indicators. This poses the challenge of developing an instrument or instruments that can 
meet local needs as well as be applicable across the many programs. The next step was to obtain the 
research partners’ opinions about the possibility of developing indicators associated with each of 
the seven possible bases, with a view to ordering the perceived value of the bases. 

Identification of indicators 

Data from 46 paid and volunteer coordinators, paid teachers, and volunteer tutors were obtained at 
five of the sites. The data related to the possible indicators; that is, evidence or examples of 
changed learner behaviour, for each of those seven bases. A summary of the responses is shown in 
table 9 in the support document. 



16 



Assessing and acknowledging learning through non-accredited community adult LLN programs 




There was strong support expressed for increased confidence as being a valuable outcome of 
learner participation in these programs. Respondents, generally, claimed that there were numerous 
factors that would indicate growth in confidence and that these would be evident in their 
interactions with the learners. ‘Engagement with others’ and ‘social capital’ were generally regarded 
as strong factors. However, some respondents doubted that these could be measured or observed, 
because indicative activities tended to take place outside the learning situation. As one coordinator 
suggested: “We can only assess [social capital] as much as we know’. There appeared to be more 
opportunity for this to occur in neighbourhood centres where there was more interaction between 
learners and staff, than in one-to-one tutoring. 

Ambivalence was expressed about ‘life trajectories’, with one tutor suggesting it sounded ‘a bit 
Californian’. Some respondents struggled with understanding the term ‘agency’, and the perception 
generally was that it would be difficult to identify concrete indicators. This reaction was even 
stronger for ‘personal growth’, which was described by one coordinator as an internal change and, 
therefore, far less visible than growth in self-confidence, for which it was suggested there were 
observable behavioural changes. Nevertheless, one person with significant experience in 
community settings believed indicators for all the bases could be identified by developing 
appropriate scenarios to which learners could respond. 



On the basis of the ratings given by the 46 respondents, and from the responses in the interviews, 
five of the bases identified from the research literature were proposed as foundations for 
developing draft instruments for assessing outcomes: self-confidence and personal competence; 
engagement with others; attitudes to learning; life trajectories; and social capital. It was concluded 
that identifying a number of possible bases would ensure that the partner organisations were not 
restricted in their choices of wider learning outcomes when attempting to assess learner outcomes. 

The researchers worked with the partners to develop this ‘tentative portfolio’ of possible 
instruments into a ‘working portfolio’ that would become the basis for the trial. The steps in the 
development of this working portfolio are summarised from left to right in table 2. The first step in 
this phase was to canvass views on a range of possible monitoring instruments. 

Responses to possible instruments 

During the literature review phase, examples of instruments used in assessing a range of wider 
learning outcomes were identified, not all in the literacy and numeracy field. For example, some 
were instruments for identifying ‘soft outcomes’, described by Dewson et al. (2000, p.2) as: 
‘outcomes from training, support or guidance interventions, which unlike hard outcomes, such as 
qualifications and jobs, cannot be measured direcdy or tangibly’. 



Table 2 Development of working portfolio of instruments 



Initial 


Outcomes from 


Testing 


Selection of 


Modification of 


Trialling of 


exploration 


review of 


outcomes from 


bases and 


selected 


instruments 




research 


review of 
research 


instruments 


instruments 




Review of 


Seven possible 


Interviews and 


Five bases for 


Five selected 


Six instruments 


research 


bases for 


discussions with 


monitoring 


monitoring 


trialled by five 




monitoring 


coordinators, 


outcomes 


instruments 


partners in four 




outcomes 


teachers, tutors 


selected 


modified from 


states 




identified 

Nine examples of 
monitoring 
instruments 
identified 


and learners 


Five examples of 
monitoring 
instruments 
selected 


outcomes of 
interviews and 
the selected 
bases and one 
composite 
assessment 
instrument 
developed 
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As shown in table 2, nine examples of assessment instruments were initially chosen for 
consideration by coordinators, teachers and tutors at the research partner sites. These instruments 
were characterised only as examples that had been found during the literature review and were 
selected in order to provide a range of the types of instruments that could be used, rather than 
necessarily being considered for final adoption. Respondents were asked to identify the potential 
usefulness of each instrument for assessing the wider outcomes of learning in non-accredited 
language, literacy and numeracy programs, on a scale of 1 (not useful) to 5 (very useful). 

On the basis of the respondents’ ratings (see support document), and from comments in the 
interviews, it was decided to focus on the five monitoring instruments most favoured. An 
additional instrument was created in order to provide the partners with another comprehensive 
instrument from which coordinators might choose. The six instruments, which are attached as 
appendix 2, are: 

A Changes in confidence 

B Learner outcomes: personal, social, economic 
C Individual learner profile 
D Learner indicators of success 
E Criteria for achievement 
F Skills and wider outcomes. 

The instruments 

Each of the instruments A to E is an adaptation of the five instruments selected through the 
process shown in table 2. Instrument A was adapted from the Catching Confidence grid (Eldred et 
al. 2004) and provides different contexts for learners’ confidence and participation. Instrument B is 
an adaptation from Foster, Howard and Reisenberger (1997) and allows literacy tutors and learners 
to provide examples of outcomes in three categories: personal, social and economic. Instrument C 
is adapted from individual profiling using soft outcomes indicators developed through the 
European Social Fund and adapted by the Welsh Department for Work and Pensions (2003) and 
provides for ranking of learner perceptions under three headings: achievements, social, and 
personal growth. Instrument D is an adaptation of Eldred’s (2002) list of indicators of success 
identified by literacy learners, with the addition of a ranking scheme which provides for charting of 
progress. Instrument E is based on Charnley and Jones’s (1979) ‘emergent criteria’ of literacy 
achievement, and includes selected items from their lists of indicators in five categories: personal, 
social, socio-economic, cognitive and enactive achievements. Trial instrument F is a composite 
instrument developed by the researchers from their review of research and the interview responses 
and provides for examples under four headings: skills outcomes, personal outcomes, social 
outcomes and vocational outcomes. 

The intention was to provide a range of instruments, any of which might be utilised to meet the 
requirements of a particular program or particular learners. The instruments per se were not under 
scrutiny, rather their capacity to assess outcomes not captured by existing instruments. 
Consequently, in response to suggestions made during the discussions about outcomes and 
indicators, some minor modifications to the wording in the draft instruments were made. Inevitably 
and importandy, there are some overlaps of groupings and indicators across the instruments. 
However, it was not intended that the instruments would be mutually exclusive; rather, that each 
should have a slighdy different emphasis in assessing outcomes, which would give coordinators 
some leeway to meet particular needs. 

It was also intended to provide examples of instruments that assessed learning outcomes beyond 
language, literacy and numeracy competence, which would therefore not usurp either existing 
instruments that measured language, literacy and numeracy skills or the autonomy of the 
coordinator. However, because of the need for variety in the instruments selected from the range 
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available, two of the instruments drawn from literacy contexts, instruments C and D, and the 
expanded version, instrument F, include such indicators. 

No attempt was made to validate the instruments or any modifications to them during the trial 
period, since that was not part of the research purpose. The instruments trialled were simply 
examples of a range of instruments drawn from the research literature in order to determine whether 
such instruments had the potential to identify wider learning outcomes in non-accredited adult 
language, literacy and numeracy learning. 



Trialling the instruments 

The trial period ran from late May to early July 2007. It had been hoped to use the instruments in 
pre-test and post-test situations, in order to document the extent of learner progress, but in most 
instances there was not sufficient time for this within the constraints of the project and the 
partners’ program schedules. 

A workshop was held in Adelaide in July 2007, where representatives of the partner organisations 
reported on the outcomes of the trials. Edited versions of the reports are included in the support 
document. 

All of the partners, except one, distributed all six instruments to the tutors and invited them to 
choose the instrument/ s they saw as most appropriate. In Western Australia, where the 
administration of the statewide volunteer groups is centralised, two senior Read Write Now! staff 
and a regional volunteer coordinator selected two instruments to send out to the groups. A variety 
of approaches was adopted by the tutors at all sites, with some at one organisation opting to use all 
six instruments with their learners over the trial period. 



Table 3 Choice of trial instruments at each site 



Partner organisation 


Final choice 


Comment 


Aberfoyle Community Centre (SA) 
Glenroy Neighbourhood Centre 


Instrument D 

Instrument B - Road rules 


Tutors could choose any of the six 
instruments. 


(Vic.) 


Instruments C and D - Women’s 
ESL 

Instrument C - Story group 

Instrument F - African Women’s 
sewing class 

Instrument E - non-accredited 
General Certificate in Education 
learner 


Tutor also devised own instrument. 

Tutors could choose any of the six 
instruments. 


Hackham West Community Centre 
(SA) 


Instrument C 


Initially instruments C, E, F chosen 
by tutors for trial. 


Pine Rivers Neighbourhood Centre 
(Qld) 


Instruments A and C 


Tutors could choose any of the six 
instruments. 


Read Write Now! (WA) 


Instrument E modified* 


Initially instruments C and E chosen 
centrally for trial; language slightly 
modified. 



Note: ESL = English as a second language 

* See appendix 3 for a copy of the modified instrument. 



Table 3 shows that no single instrument stood out as a preferred choice amongst the partners. 
Instrument C (Individual Learner Profile) was the one most identified as potentially useful, with 
initial nominations from four of the five partners. However, at the end of the trial only two 
partners nominated instrument C as their primary choice, and one of those co-nominated it with 
instrument A (Changes in Confidence). 
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Issues arising from trial of instruments 

The reports and subsequent discussions at the workshop identified a number of issues in relation to 
the use of such instruments in non-accredited adult language, literacy and numeracy programs. By 
far the biggest issue was the language used in these instruments. There was general agreement that 
much of the language used in the trial instruments was too difficult for many learners in these 
programs, particularly English as a second language learners. It was concluded that terms should be 
either simple enough for the learners to understand themselves or to enable the tutor to explain 
them easily to the learner. There was also the suggestion, incorporated in one of the modifications, 
that a key term such as ‘confidence’ could be briefly defined at the top of the instrument. Some felt 
there might be a case for different instruments for different types of learners. 

Related discussion focused on the use of such terms as ‘soft outcomes’ and ‘social capital’ and 
whether practitioners should be looking for terms that are better understood by policy-makers and, 
by implication, funding bodies. One suggestion was that terminology relating to community 
capacity-building outcomes should be adopted. It was suggested that the number of questions 
should be determined by the need for all learners to be able to identify those that applied 
particularly to them. Generally, there was support for using scales with words (for example, ‘about 
the same’, ‘a little better’) rather than numbers. 

Workshop participants agreed that the instruments should be completed joindy by the tutor and 
learner. Such a procedure appears to encourage the development of confidence gained from 
completing an instrument, as well as assist in tailoring the tutoring to meet an individual learner’s 
needs. One concern raised was the possible additional workload for volunteer tutors, with a couple 
of the coordinators reporting some resistance from tutors to taking part in the trial for that reason. 
The audience for the completed instruments was seen as primarily the tutor and learner, and 
subsequently the program coordinator (where relevant), and possibly managers of these providers. 
However, it seemed that managers were more likely to be provided with a summary of the 
achievements of all learners rather than individual assessments as recorded by the instruments. 

One question that had been raised earlier by the researchers was whether learners should be 
awarded a certificate on the basis of achievement as recorded by the instruments. This question 
generated a variety of responses from coordinators in the discussion. For example, one said the 
practice had been to present certificates of participation to all learners annually but that had been 
discontinued, because some learners, particularly those with acquired brain injuries, claimed they 
did not deserve them and had questioned the value of them. Another centre coordinator suggested 
that a certificate of achievement implies a set of hard outcomes, and that might encourage a learner 
to think they had finished learning, whereas in a course such as obtaining a driver’s licence, ‘you can 
usually identify that they have further literacy and numeracy needs, not just getting their licence’. 

At another centre, there was strong support from tutors and learners for the acknowledgement of 
this learning in the form of certificates, including learners with intellectual disabilities who valued a 
piece of paper that indicated that they were regular attendees. It was suggested that certificates of 
achievement could be based on the growth of self-esteem and confidence, and that they were 
particularly important for learners from backgrounds where learning was not rewarded and where 
previous education had not been a positive experience for them. 
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Discussion 



An abiding concern and focus of government-sponsored policy initiatives and practices associated 
with vocational education and training in Australia has been securing uniformity in its purposes, 
processes and outcomes, which has been achieved through the establishment of national standards 
of competence for and across industry sectors, nationally consistent training packages, and an 
assessment focus on reliability of outcomes through the use of objectives that are measurable and 
through processes that acknowledge or deny competence. The National Reporting System is part of 
this approach. However, this project reveals that, in the area of assessing the wider learning 
outcomes of learners’ participation in non-accredited language, literacy and numeracy programs, 
diversity rather than uniformity is the reality and central concern for the purposes, processes and 
outcomes of learners’ learning. 

Consequently, the approaches to assessing and acknowledging the outcomes of non-accredited 
language, literacy and numeracy learning need to be diverse and open to benchmarks or goals of 
different kinds. Some of the factors associated with the diverse purposes, processes and outcomes 
which were identified in the research are discussed below. 



Purposes and processes of assessment 

Factors associated with the purposes of the assessment of unintended learning outcomes include 
the diversity of learners’ backgrounds, the level of their language skills and the potential for them 
and their often voluntary tutors in using the instruments. Indeed, the competence and confidence 
of these tutors in choosing and completing the instrument and their capacity and confidence in 
assessing learners’ development was a second key factor. A third factor is the extent to which any 
selected instruments might be seen as an initial, formative and/ or summative assessment 
instrument, and the purposes for which the instruments are being used. Some of the factors 
associated with both learners and tutors, together with the implications for using the instruments, 
are set out in table 4. 



Table 4 Factors associated with the process of assessment 



Focus 


Factors 


Implications for instrument use 


Learners 


Readiness - level of language competence 


Complexity of instrument 


Learners 


Readiness - level of confidence 


Capacity to engage and kinds of outcomes desired 


Learners 


Readiness - need for tutor assistance 


Requirement for one-on-one assessment and 
support 


T utors 


Readiness - level of language competence 


Capacity to understand, select and explain the 
instrument’s worth to learners 


T utors 


Readiness - level of tutoring competence/ 
confidence 


Capacity to tailor/modify to learners’ level of 
readiness 


T utors 


Workload 


Length and complexity; additional demands on time 


T utors 


Volunteer 


Difficulties for moderation; role of coordinator 



Table 4 shows some of the variables associated with the process of assessment and factors related 
to the design and administration of appropriate instruments. For instance, the capacity of learners 
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to engage — either on their own or with their tutor — with the assessment processes and any 
particular assessment instrument (their ‘readiness’) will be premised upon their level of language 
competence. Similarly, the level of the tutors’ competence to engage in the assessment process, 
including the selection of the most appropriate instrument, is likely shaped by their competence and 
confidence. Further, the tutor’s capacity to use the instruments effectively and modify them to suit 
particular individual or cohorts of learners’ needs shapes the assessment process and its outcomes. 
Also, given the voluntary role of the tutors, there are limits to what their workloads and 
commitments can withstand. So, through consideration of these kinds of factors, the design, use 
and outcomes of assessments need to be understood. In the following sections, these diverse 
factors are elaborated. 

Diversity of learners and their purposes 

Given the findings from the international and Australian research discussed earlier, it was 
anticipated that catering for the assessment of the diversity of learners in non-accredited language, 
literacy and numeracy would be a key challenge in developing instruments for assessing and 
acknowledging learning outcomes. The first research question for this study: ‘What are the main 
language, literacy and numeracy learner groups and their needs in non-accredited programs?’ was 
designed to confirm the extent of the diversity. A re-examination of selected data from the 2006 
survey of Australian providers (Dymock 2007a) found that, nationally, about two-thirds of such 
learners are women, that the age range is broad, with the majority being in the 30—49 years bracket, 
where work and family demands are likely to be particularly strong, and that there are numerous 
reasons for seeking support. The most frequently reported motivations perceived by the responding 
providers are related to social and personal development needs, but often a strong vocational 
element was also reported. This finding was reinforced by data collected from the interviews for the 
present project, conducted at six different sites in four states, which confirmed Dymock’s 
assessment of learners in non-accredited language, literacy and numeracy as: 

lacking in confidence, often with negative experiences of school; some have disabilities, 
others are grappling with English as a second language; sometimes they are already in 
employment which is low skilled (but with exceptions); and sometimes they are already in 
accredited training but are not coping. (Dymock 2007b, p . 1 5) 

So, underpinning concerns about confidence are issues of circumstance, health, age, place of birth 
and other factors which have inhibited language development and with it, the confidence of the 
learners to use English language in the various situations in which they find themselves. 

In this study, catering for the range of needs and outcomes is well illustrated by the responses from 
the tutors in the Glenroy Neighbourhood Centre program, where each tutor selected the 
instrument they thought best suited the learners in their class. The result was that five different 
instruments were selected; one tutor designed another instrument. On the other hand, three of the 
partner organisations decided on one particular instrument as best meeting their needs. 
Nevertheless, in relation to diversity, it is significant that, across the partner organisations, each of 
the particular instruments chosen is different. In other words, individual and/ or group needs 
predominate as the basis for selection of an instrument. 



Level of language 

The level of language was the most frequent issue nominated during the trials of the instruments. 
This determined the readiness of learners to participate in the assessment processes and was 
perceived to be a particular issue for those who had English as a second language. The level of the 
language used in particular instruments was considered too difficult for learners at some sites. 
Elsewhere, tutors used terms like ‘convoluted’, ‘obscure’, ‘academic’ and ‘universityfied’. The issue 
then becomes one of the lowest common denominator: should the design of the instruments be 
based upon the likely lowest language competence of the tutors and their learners or on variations 
which permit their broad use, yet offer levels of more nuanced use? 
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Issues of learner capability and the capacities of tutors to use the instruments therefore need to be 
considered as central elements in the design and administration of processes that assess and 
acknowledge these kinds of learning. 

Tutors’ role and abilities 

In most cases, the trial instruments were reported as being completed jointly by the tutors and 
learners. There was variation in tutor input in explaining the purpose of the instrument to the 
learner, in clarifying any obscure terms or difficult words, and in helping the learner decide on the 
most appropriate response. The implication here is that the tutor needs to be clear on the purpose 
of the instrument as a means for monitoring outcomes; the tutor also needs to understand the 
terms and concepts used in the instruments and to be as objective as possible when helping the 
learner to choose appropriate responses to the items on the instruments. 

An issue arising here is that, just as there is a diversity of learners in non-accredited language, 
literacy and numeracy, the volunteers themselves are diverse in their backgrounds, levels of 
language and literacy competence, and instructional and assessment competence. Some tutors 
were often not confident of their understanding of some of the terms and concepts and/ or with 
their ability to explain them, which may affect their capacity and willingness to administer certain 
types of instruments. Clearly, these issues are the inescapable reality in many of these non- 
accredited programs and need to be seen as central issues in the design and administration of 
assessment instruments. 

The voluntary aspect of tutoring in the program also needs to be considered as a further crucial 
element. Coordinators indicated that, since tutors volunteered their time to assist learners, being 
asked to administer an instrument they didn’t support or felt uncomfortable using might be seen as 
an imposition, particularly if additional training were also required. Gaffney and Humm (2002, cited 
in Grief & Windsor 2002, p.6) report concerns from community organisations in Britain that the 
development of ‘measurement systems’ would turn volunteers into ‘professionals’. However, the 
positive outcomes of the process in the present study, discussed below, suggest that the use of an 
instrument might be integrated as part of the tutoring role and as another way of further developing 
the learners’ conception of themselves as learners. This would also prevent tutors being seen as 
monitoring and summatively assessing learners. Encouraging such attitudes would need to be 
incorporated into tutor training courses. 

Use of instruments 

The final issue for discussion is the place of the selected instrument within the language, literacy 
and numeracy program. As seen in the trials, the choice of instruments will be determined in part 
by whether they should be used formatively to provide a means of reporting outcomes over time 
(that is, monitoring progress), or whether they should be used a summative assessment instrument, 
or as some combination of the two. 

The norm in non-accredited language, literacy and numeracy for initial assessment of a learner’s 
needs and capabilities tends to take place through interview and discussion, so it seems unlikely that 
the sorts of instruments trialled would be used for initial assessment by the coordinators. Several of 
the coordinators commented that such instruments might create unhelpful anxiety for potential 
learners, who are often at a quite vulnerable stage and who may view it as a test of their inadequacy. 
Overall, these instruments are likely to be best utilised at the tutor— learner level as part of teaching- 
learning interactions. 

In addition, there is the question of whether a particular instrument can be used in conjunction 
with an existing assessment instrument. The use made of instruments C and D in the trial suggests 
that there may be a place for a combined instrument; one partner indicated an intention to integrate 
the chosen instrument with an existing assessment instrument to develop a single formative 
reporting instrument. However, as noted earlier, the purpose of this study was not to develop 
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specific indicators of language, literacy and numeracy skills or of wider learning outcomes, but to 
explore whether assessment of wider learning outcomes was possible. More research, consultation 
and trialling would be required for the development of a set of comprehensive indicators. For 
language, literacy and numeracy skills, the indicators of competence used in the National Reporting 
System would be a possible starting point in further research, but no attempt was made in this 
project to make comparisons with such indicators. 



Other issues 

Other issues arising from the research findings include whether the awarding of certificates would 
be beneficial to learners, the extent of further reporting of the outcomes (for example, to a 
management and/ or funding body), and the implications for reporting outcomes nationally. Table 5 
summarises the implications of the various factors associated with these other assessment outcomes 
of assessment. It identifies the different foci for assessment, the kinds of factors which need to be 
considered and the implications for assessment instruments and their use. 



Table 5 Factors associated with outcomes of assessment 



Focus 


Factors 


Implications for instrument use 


Reporting 


Kinds of goals reported on 


Range of goals included 


National reporting 


Consistency 


Training of tutors; moderation by coordinators 


Learner involvement 


Capturing worth of learner involvement 
and engagement 


Reporting on outcomes beyond those listed 


Certificates 


Purposes of certification and subjects 
requiring certification 


Degree of measures required (if at all) to 
validate and make certification worthwhile 



Other outcomes for learners 

One of the unexpected outcomes of trialling the instruments was the positive response of tutors 
and learners to the assessment process itself. There were reports from the participating tutors of 
learners developing confidence as a result of participation in the assessment tasks, regardless of the 
actual results. The fact that completion of the instruments was almost invariably undertaken jointly 
by the tutor and learner may well have contributed to this outcome, with the tutor supporting the 
learner in responding to the items. However, tutors also reported benefits, not merely as a 
consequence of learners achieving outcomes, but also from engaging in the process with their 
learners. For example, the tutor in the ‘Road rules’ program at Glenroy Neighbourhood Centre 
observed how the interactions resulting from responses to the instruments led to a broadened 
discussion within the group. Further, the tutors at the Pine Rivers Neighbourhood Centre were 
reported to be stimulated to ‘re-examine the what, why and how of their own tutoring’. 

Importantly, the instruments then became a means to appraise learning outcomes and also a way to 
support that learning by explicitly drawing attention to the kinds of development that the learners 
might be receiving through their participation in these programs. Given the nature of learners’ 
participation and also that of the tutors, the worth of unanticipated outcomes should not be 
underestimated. Nevertheless, because they are unanticipated, it may be impossible to include 
sufficient provision for them on any instrument. 



Acknowledgement of learning 

There is no clear or single answer to the question of whether learners should receive a certificate of 
achievement on the basis of outcomes indicated through the assessment process. The value of 
awarding such certificates was perceived to be quite different across both participating organisations 
and learner cohorts. It seems that some learners, in particular those with an intellectual disability, are 
wary of acknowledgement that may be viewed as patronising. One coordinator was concerned that 
such an award might be interpreted by learners as the end of their need for learning, a perception he 
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did not want to encourage, for the learners’ sake. Yet, conversely, there was strong support in at least 
one partner organisation for certificates, including for intellectually disabled learners to acknowledge 
regular attendance, and generally for learners because it was felt that achievement should be 
recognised. Here again, diversity in purposes and processes is determined by the needs and 
perceptions of both learners and tutors participating in these programs. 

An issue for the awarding of certificates would be ensuring consistency of reporting by tutors. This 
presumably would be part of the training in the use of the instrument. There might also be a need 
for a moderating role by the coordinator. 

Reporting 

Given that in the trial the instruments were used to identify individual achievement and as a means 
for tutors to identify areas of strengths and weaknesses, both in learning and teaching, it seems that 
the greatest value of the instruments lies at the tutor— learner level. Results could also usefully be 
conveyed to a coordinator, where applicable, to enable the coordinator to review progress and 
discuss this with the tutor if necessary. External reporting (for example, to management or a 
funding body) would best be done through the aggregation of individual assessments of outcomes, 
the nature of that summary depending on the instrument used. Importandy, however, these 
instruments provide a means to identify, elaborate and potentially qualify and quantify important 
learning outcomes emerging from participation in these programs, which need to be acknowledged 
and rewarded. The question of national reporting is discussed under research question four, below. 



Research questions 

Research question 1 

A response to the first research question: What are the main language, literacy and numeracy learner 
groups and their needs in non-accredited programs?’ is provided above in the section on ‘Diversity 
of learners’. 



Research question 2 

What are the main outcomes from non-accredited language, literacy and numeracy programs, and 
what is the role of personal agency and goals in achieving these outcomes for individual learners? 

In regard to the wider benefits of learning, it was not difficult to identify outcomes in the interviews 
at the partner sites under all seven categories identified from the review of literature (self- 
confidence; engagement with others; attitudes to learning; agency/ pro-activity; life trajectories; 
personal growth; and social capital). This ease of categorisation confirms the need to consider the 
‘wider benefits’ of learning, that is, beyond the development of language, literacy and numeracy 
skills. It also confirms the findings reported in the literature (Ward & Edwards 2002; Westell 2005; 
Swain 2006) on the impact of learning on learners in adult literacy and numeracy programs. 

Furthermore, the findings are consistent with recent research and policy developments in Britain 
vis-a-vis the extent of ‘soft outcomes’ in basic skills and workplace programs. However, it is worth 
noting Butcher and Marsden’s criticism of what they saw as the narrowness of linking ‘soft’ 
outcomes with ‘hard’ ones. This is because they believed: 

. . . such an approach narrows the interpretation of social inclusion to employability and 
educational achievement and ignores vast swathes of the community and voluntary sector 
working with clients who may never access jobs or educational qualifications. 

(Butcher & Marsden 2004, p.6) 

Similar concerns were expressed by some respondents to Dymock’s (2007a) survey of Australian 
providers of non-accredited learning in adult language, literacy and numeracy. 
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Looking at the role of personal agency, the findings here indicate that the development of an 
individual ‘learning identity’ may be a key element of engagement in the learning process. Many of 
the bases for the learning associated with language, literacy, and numeracy need to be understood 
from the individual’s perspective. The degree to which individuals engage in, or withdraw from, a 
particular task will influence what they construct and how they engage with that practice. In what 
Watters and Turner (2001, p.59) described as the ‘diversity and complexity of the learners’ 
purposes’ and the outcomes confirmed through the present study, subjectivity stands as a common 
defining factor. This is consistent with what is emerging as a set of alternative ways of assessing 
learner outcomes through language, literacy and numeracy programs (for example, Eldred et al. 
2004; Watters & Turner 2001; Ward & Edwards 2002; Swain 2006). In addition, Waterhouse and 
Virgona (2005, p.29) concluded from a study of people who had succeeded, despite the apparent 
handicap of inadequate literacy, that ‘literacy issues are about identity as much as [about] skills’. The 
findings here confirm that personal agency and goals are central to successful learning outcomes in 
non-accredited language, literacy and numeracy programs. However, the nature of ‘learner identity’ 
and how it might be enhanced need further exploration. 



Research question 3 

How might these outcomes best be ‘measured’ in order to indicate the extent of progress that 
learners are making, in terms of reporting against: existing frameworks; personal needs and interests 
of participants; and making a comparison between the two? 

The trials showed that each of the six instruments is capable of making some assessment of the 
wider benefits of learning, but that the level of language used needs to be appropriate to enable 
learners to understand and/ or to enable the tutors to interpret for them. Although all of the 
instruments had the potential to identify changes in individuals since an earlier previous report/ 
assessment — some more clearly than others (for example, A, C and D) — there was insufficient 
time in this study to determine whether the instruments were capable of recording ‘progress’. 

Furthermore, the relative value given to each of the wider benefits and, therefore, to the choice of 
indicators, will vary at a local level, so a ‘portfolio’ of instruments is necessary. This allows them to 
be adapted to local contexts. As noted, it was not the intention of the research to monitor the 
development of language, literacy and numeracy skills, although some instruments did incorporate 
those aspects because of their educational origins; it was also not the purpose of the research to 
identify or validate any particular instrument or to compare the extent to which each of the sample 
instruments was capable of monitoring outcomes in each of the areas identified. Instead, the 
concern was to identify a range of approaches capable of undertaking this task. 

The assessment process seems to work well when it is a joint activity between learner and tutor, and 
there are potential benefits for both learner and tutor from engagement in that process, beyond 
what is included in the instrument. Whether such instruments are used instead of or in conjunction 
with existing frameworks is a decision for those responsible for the program, but their 
administration is likely to be formative and/ or summative, rather than for initial goal-setting. 

Research question 4 

To what extent might such indicators of progress be appropriate for incorporating into a national 
reporting process? 

The need to allow for a wide range of individual learning requirements and outcomes and for 
coordinator and tutor preference for particular instruments point towards the answer to the final 
research question. As indicated, it was not the intention of this project to develop a specific and 
broadly applicable instrument or to compare possible indicators of outcomes with those developed 
for the National Reporting System. The kinds of learning being considered here are not readily 
accommodated by measures of the kind deployed in the National Reporting System. In this context 
it is worth revisiting the warning given by Grief and Windsor (2002, p.63), quoted in the review of 
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research, that attempting such correlations risks losing ‘the very features of non-accredited learning 
for which this option is currently valued’. 

The findings suggest that the development of a single instrument to facilitate consistent national 
reporting of learner outcomes in non-accredited language, literacy and numeracy learning would 
require considerably more research and consultation and, as noted by Grief and Windsor, has the 
potential to imperil the very purposes it seeks to achieve. As discussed above, even among the five 
partner organisations, there was no consensus about which instrument was most appropriate, and 
within organisations there was considerable diversity in the selection of instruments by tutors. 
Nevertheless, the identification of five possible bases for considering the wider benefits of learning 
and the alacrity with which the organisations and many of the tutors accepted the sample instruments 
for trialling suggest that many practitioners would be open to developing and trialling more 
rigorously validated instruments. A development such as this could lead to improved learning plans 
for individuals, more satisfying teaching for tutors, and consistent reporting to funding bodies. 



Conclusion 

The diversity of learner needs, motivations and outcomes in non-accredited language, literacy and 
numeracy programs, as well as the processes of tutoring and learning, means that a choice of 
instruments for assessing and acknowledging learning is necessary. This study has shown that it is 
possible to develop instruments to assess and acknowledge the wider benefits of learning. These 
instruments can provide options to cater not only for the learners’ individual needs, but also for 
tutors’ and coordinators’ abilities and preferences. 

The findings suggest that there are potential benefits for both learner and tutor from joint 
engagement in the assessment process, but tutors are likely to need training to better understand 
the purposes and language of the various instruments. The language of the instruments also needs 
to be intelligible to the learners, particularly those for whom English is a second language. Overall, 
despite criticism of some of the language and of aspects of particular instruments, there was very 
positive acceptance of the intention of the instruments — what one tutor group referred to as 
‘having a measure of achievement not couched in the language of training packages and 
competency-based training’. 

Further research is therefore needed into the desirability and practicality of developing a scheme for 
national reporting of the wider outcomes of learning in non-accredited community language, 
literacy and numeracy. At the individual level, the development of learner identity through the 
learning process needs further exploration in non-accredited adult learning generally. 

In their study of the link between learning and confidence, Eldred et al. concluded: 

Without increases in confidence, many adults will remain non-participants, not achieving their 
full potential in personal development as well as skills and qualifications. The importance of 
non-threatening first-step learning which gives learners time to gain confidence, [and] develop 
their identity as successful learners, and supports diverse aims and aspirations is vital. 

(Eldred et al. 2004, p.57) 

Community education providers are key agents in the provision of ‘non-threatening first-step 
learning’ in Australian education and training. This research has shown that it is possible to assess 
the wider benefits of the non-accredited community language, literacy and numeracy learning 
provided by many of these organisations and, in so doing, better acknowledge the significance of 
the contribution that individual learners make to the social and economic development of their 
local communities and, ultimately, of the nation. 
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Appendix 1: 
Cover email letter to partner 
organisations with trial instruments 



Dear Debbie, John, Jacinta, Tamara, Marcia and Liz 

As promised, attached is a ‘portfolio’ of six possible instruments for measuring progress in non- 
accredited language literacy and numeracy, for you to trial with your tutors and learners. These have 
been developed following interviews and discussions in four states arranged through the six 
partners with coordinators, teachers, tutors and learners. 

Originally our plan was to send these instruments for your responses, and then for us to fine tune 
them before sending them to you again. However, given the extensive consultations that have taken 
place over the last two months, and the time constraints of the project, it seems more expeditious 
to send them to you as drafts for trialling, but with the proviso that you can amend these 
individually for ‘local’ use if you wish, as well as create your own instrument to add to the portfolio, 
if you prefer. 

Here is what we would like you to do: 

1 Consider which of these draft instruments, if any, might be appropriate for monitoring the 
progress of learners in your language, literacy and numeracy program. 

2 If necessary, feel free to modify the wording slightly to suit your own circumstances. You may 
also want to create a completely new instrument to test in your program, now that you have 
seen what these are like.* 

3 Trial these instruments with as many tutors and learners as possible between now and mid July. 

4 Document the feedback from tutors and learners on the instruments you select for use, along 
with any changes you have made, and report on these at the workshop in Adelaide on 20 July. 

5 Among the questions you might address are: 

How can these instruments best be used in your practice? 

What are their relative strengths and weaknesses? 

Are there any particular preferences for instruments, from learners and/or tutors and/or 
coordinators and if so, why? 

In the trial, were they completed by learners, tutors or a combination? 

If you had to select just two of these instruments, which would they be and why? 

Are there issues of monitoring progress in non-accredited language, literacy and numeracy which 
are not met by any of the instruments in the portfolio? 

To what extent might it be possible to adopt one or more of these instruments for national use? If 
this is a feasible idea, how might that be done? 

Should learners be awarded a certificate of achievement on the basis of progress made? If so, on 
what basis and over what time period should this be awarded? 

*One suggestion made to us was the possibility of developing a number of scenarios against each of 
the possible bases for monitoring progress, e.g. for ‘Confidence’, the scenario might be: “You have 



NCVER 



29 




been dropped in the middle of [city]. Would you be able to get home from there, and if so how?’ 
We think this notion is worth considering, but that the scenarios would best be developed locally 
rather than genetically by us, in order to meet individual needs. 

The original intention of this project was to see if it is possible to identify wider outcomes, i.e. those 
beyond the development of language, literacy and numeracy skills. Some of the attached 
instruments maintain that focus, and might be used in conjunction with skills assessments you 
already use; some of them incorporate both skills and broader outcomes. 

Ideally, ‘progress’ is measured between two points, so it would be best if you could trial these over 
at least two sessions, say a month apart. However, if that is not possible within the time available, 
even a single session should help determine if an instrument is appropriate or not. 

Please contact us if you have any concerns or queries. We will share these and our responses where 
appropriate with the other partners in order to keep you all involved and maintain this as a 
consultative process. 

Darryl will give you a call within the next couple of weeks to see how you are going with 
implementing the trial. 

We would appreciate it if you could acknowledge receipt of the portfolio, and perhaps give us some 
indication of how you are likely to trial the instruments over the next eight weeks. 

With best wishes for a successful trial period, and our thanks for your ongoing cooperation, 



Darryl Dymock and Stephen Billett 

Griffith University 
18 May 2007 
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Appendix 2: 
Trial instruments for 
monitoring progress 



Trial Instrument A: Changes in confidence 

Learner name: 



This report is for the period: to 

Write a number between 1 and 4 in each box below to indicate how confident you feel since 
starting your adult literacy program. Leave blank any boxes that do not apply. 



Key: 1= Not confident 
Statements 



2 = A little confident 3 = Quite confident 4 = Very confident 

At Socially -with In shops and At work 

home friends government offices 



1. I am confident when meeting new 
people 

2. I am confident that I can use what 
I learn from the tutor in daily life 

3. I am confident to speak to a 
person I do not know 

4. I am confident to speak in a group 

5. I am confident in situations which 
may be difficult 

6. I feel I am generally a confident 
person 



7. I am confident I am learning from this literacy program 

8. I am confident that I have valuable skills to offer 

9. I feel I am generally a confident person 

10. lam confident with my reading 

11. lam confident when writing things down 

12. lam confident with my spoken English 

13. lam confident working with numbers and figures 

Total confidence score: 

Tutor Name: 

PROGRESS REPORT: From the individual responses above, in your opinion do these collectively 
indicate an increased level of personal competence since the last report? YES/ ABOUT THE 
SAME/ NOT SURE (please circle one) 

Comment: 



Signed: 



Date: / / 
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Trial Instrument B: Learner outcomes — personal, social, economic 

Learner name: 

This report is for the period: to 

Indicate changes since last report. Leave blank any boxes that do not apply. 

Personal outcomes Examples/Indicators 

Confidence 

e.g. now able to catch a bus on 
their own; stands up for own 
rights; can deal with authorities; 

Personal goals, ambitions 

e.g. now wants to do TAFE 
course; wants to travel; now has 

clear goals beyond this program 

Social outcomes 

Engagement with other 
individuals 

e.g. now able to have two-way 
conversation; mixes better with 
others; reading to children 

Involvement in the community 

e.g. has joined a club; has joined 
library; now talks to child’s school 

teachers 

Economic outcomes 

Employment 

e.g. got a job; started checking 
job adverts; attended job 
interview; working as volunteer 

Life management 

e.g. ability to shop 
independently; taking charge of 
own affairs; taking responsibility 
for improving health 

Learning 

e.g. took another course; is now 
aware of own strengths; has 
obtained learner’s or driver’s 

licence 

Other outcomes 



Tutor Name: 

PROGRESS REPORT: In your opinion do the individual responses shown above collectively 
indicate an increased level of personal competence since the last report? 

YES/ ABOUT THE SAME/ NOT SURE (please circle one) 

Comment: 

Signed: Date: / / 
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Trial Instrument C: Individual learner profile 

Learner name: 

This report is for the period: to 

Please tick one box in each row. 

1 2 
Never Rarely 

Achievements 

1 I am confident about my writing 

2 I am confident about my reading 

3 I am confident about my spoken 
English 

4 I am confident with my numeracy 

5 I am aware of my strengths and 
weaknesses 

6 I can complete official instruments 

7 I am able to learn new things 

8 I can find my way around and use 
public transport 

9 I can use a computer for writing 

Social 

10 I am good at listening 

11 lam confident when meeting new 
people 

12 lam confident to speak in a group 

13 lam confident in strange situations 

Personal Growth 

14 I know what I want to do in my life 

15 I feel good about myself 

16 I am able to speak up for myself 

Tutor name: Total score: 

PROGRESS REPORT: In your opinion do the individual responses shown above collectively 
indicate an increased level of personal competence since the last report? 

YES/ ABOUT THE SAME/ NOT SURE (please circle one) 

Comment: 



3 4 5 

Sometimes Often Always 



Not 

applicable 



Signed: 



Date: / / 
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Trial Instrument D: Learner indicators of success 



Learner name: 

This report is for the period: to 

Please tick one box in each row to show how much you think you have improved since starting this 
adult literacy tutoring. 



Achievement 


About the 


A little 


Much 


Not 


Does not 




same 


better 


better 


sure 


apply 



1. I can read a newspaper or magazine 

2. I can read instruments and letters 

3. I can write letters & cards at home 

4. I can read at work 

5. I can write at work 

6. I can use the phone to speak to 
people in authority 

7. I can read things and look things up at 
home 

8. I can read signs 

9. I have increased confidence and feel 
better about myself 

10. lam more independent and I can do 
more things outside my home 

1 1 . People in my family or at work notice that 
my reading and writing are better 



Tutor Name: 

PROGRESS REPORT: In your opinion do the individual responses shown above collectively 
indicate an increased level of personal competence since the last report? 

YES/ ABOUT THE SAME/ NOT SURE (please circle one) 



Comment: 



Signed: 



Date: / / 
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Trial Instrument E: Criteria for achievement 



Learner name: 

This report is for the period: to 

[Can be completed by learner or tutor or both, with slight variations to wording.] 

Tick all the 
achievements 
that apply 

Personal achievements 

I am more confident in myself 
I am less anxious 
I stand up for my own opinions 
I am aware of own learning achievements 
I am willing to ask for help when I need it 



ACHIEVEMENTS 

as result of participation in literacy program 



Social achievements 

I have better relationships with family members 
I can start conversations 
I have good listening skills 
I have joined a club 

I have developed relationships beyond family 
I have joined a library 

I am less anxious about meeting new people 
I can speak in a group 



Socio-economic achievements 

I have improved my chances of getting a job 
I have enrolled in another course 
I have a better understanding of the value of education 
I am now exploring job opportunities 
I have been promoted at work 
I am able to work unsupervised 
I have gained a driver’s licence or learner’s permit 
I have attended job interviews 
I am now working as a volunteer 
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Cognitive achievements 


Tick all the 
achievements 
that apply 


1 have improved my reading ability 




1 have improved my writing ability 




1 have improved my spoken English 




1 have increased my vocabulary 




1 have increased my awareness of what is going on in community/world 





Enactive achievements 

I regularly attend the literacy program 
I use reading at work 
I read to family members 
I can write notes for my children at school 
I can write paragraphs 

I can use a computer for writing and searching for information 
I can use a telephone to ring organizations, tradespeople, doctor etc. 

I can read signs (for example: street signs, safety signs) 

I am able to take part in workplace discussions 
I am able to shop by myself 
I can speak to a doctor, dentist etc without help 
I am able to use public transport by myself 

Total number of achievements ticked: 

Tutor Name: 

PROGRESS REPORT: In your opinion do the individual responses shown above collectively 
indicate an increased level of personal competence since the last report? 

YES/ ABOUT THE SAME/ NOT SURE (please circle one) 

Comment: 



Signed: 



Date: / / 
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Trial Instrument F: Skills and wider outcomes 

Learner name: 

This report is for the period: to 

Other outcomes can be added in response to individual needs and achievements 

OUTCOMES Examples since last report 

as result of tutoring (leave blank if not applicable) 

Skills outcomes 

Has improved reading 

not at all, a little more, a lot more, not sure. (please circle most 
appropriate) 

Has improved writing 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Has improved spoken English 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Has improved numeracy 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Has better learning strategies 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other skills outcomes: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other skills outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Personal outcomes 

Has increased confidence 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Has changed life goals 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Has greater sense of personal responsibility is proactive 
not at all, a little more, a lot more, not sure. 

(please circle most appropriate); 

Has now recognized own personal growth 
not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other personal outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other personal outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 
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Social outcomes 

Engages more with family 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Engages more with people outside family 
not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Is more involved in local community 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other social outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Other social outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Vocational outcomes (to be added if appropriate) e.g. in 
employment, volunteering, further education 

Outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Outcome: 

not at all, a little more, a lot more, not sure. 

(please circle most appropriate) 

Tutor name: 

PROGRESS REPORT: From the individual responses above, in your opinion do these collectively 
indicate an increased level of personal competence since the last report? 

YES / ABOUT THE SAME / NOT SURE (please circle one) 

Comment: 

Signed: Date: 
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Appendix 3: 
Modified Trial Instrument E 



Read Write Now! Draft Tutor/Learner Feedback Form E [modified] 




Learner name: 

Tutor Name: 

Region or Coordinator’s Name: 

This report is for the period: to 

[To be completed by Tutor and Learner together.] 

FIRST For the goals you have ticked, 

CHOOSE GOALS rate your improvement for this period 

AND TICK 
YOUR 
GOALS 



Literacy 


About 


Progressing 


Achieved 




the same 







To improve my reading and/or writing at home 
To improve my reading and/or writing at work 
To improve my spelling 
To improve my maths 
To improve my spoken English 
To increase my vocabulary 

To use a computer for writing and searching for information 

(If studying ) To improve my understanding of course 
materials 

(If studying) To improve my marks for assignments 
Other areas: 



Everyday Life 


About 


Progressing 


Achieved 




the same 







Regularly attend tutoring session 



Complete my homework 

Answer the phone 

Ring organisations, tradespeople, doctor etc. 

Read signs (for example: street signs, safety signs) 



NCVER 



39 




To become more independent eg shop by myself, use 
public transport, use ATM banking etc. 

To be more confident to speak with others 
To ask for help when I need it 
To become more confident in myself 
To join a club or library 
Other areas: 



Work wise (if applicable) About Progressing Achieved 

the same 

To explore job opportunities 

To get a job or a better job 

To be promoted at work 

To gain a driver’s licence or other certificate 

To enrol in another course 

To work as a volunteer 

Other areas: 



Additional comments: 



Signed (Learner) Date: / / 

Signed (Tutor) 
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Support document details 



Additional information relating to this research is available in Assessing and acknowledging learning 
through non-accredited community adult language, literacy and numeracy programs: Support document. It can be 
accessed from NCVER’s website <http:/ /www.ncver.edu.au/publications/2021.html> and 
contains: 

"y" Introduction 

A Literature review 

Possible bases for identifying wider benefits of learning 
Perceptions of indicators 
Collaborative development of portfolio 
•y" Partner profiles 

■y" Partners’ reports on trial of instruments 
•y" Letter to partners May 2007 
Interview schedules — pilot 
Interview schedules — revised 
A References 
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